#311: deploy the corrected samples_map_lite (real Place/Date data)#319
Merged
Conversation
… fix latent place_name Array.isArray bug Deploys the isamplesorg#311 pipeline fix (SamplingEvent/SamplingSite traversal for place_name/result_time, merged in isamplesorg#318) to production data: rebuilt samples_map_lite from the SAME wide.parquet currently live on R2 (verified byte-identical: exact row counts + per-source min/max pid match against https://data.isamples.org/isamples_202608_wide.parquet before rebuilding), uploaded to R2 as isamples_202608_samples_map_lite_v3.parquet (a NEW filename — the live isamples_202608_samples_map_lite_v2.parquet is served Cache-Control: immutable, max-age=31536000, so overwriting it would leave stale/broken copies at CDN edges and in visitors' browsers for up to a year; same convention _v2 itself already used over the unsuffixed name). Rebuild results (verified before deploying): place_name now resolves for 2,263,648/6,026,242 (37.6%) located samples, result_time for 5,937,692/6,026,242 (98.5%) — both were 0% before this fix. ## A second, latent bug this surfaced Testing the corrected data against the real page (not just raw parquet) found a genuine frontend bug that was invisible until now: the four places in explorer.qmd that render place_name (`Array.isArray(placeParts) && placeParts.length > 0 ? placeParts.filter(Boolean).join(' › ') : ''`) all checked `Array.isArray()`. Observable's DuckDBClient returns Arrow LIST columns (place_name is VARCHAR[]) as an Arrow `Vector` — iterable, has `.length`, but is NOT a plain JS Array, so `Array.isArray()` is FALSE on it. Every non-null place_name was silently rendering as blank. This was masked in every deploy up to today because place_name was 100% NULL in production before the samples_map_lite rebuild above — the bug had no observable symptom until real data started flowing. Extracted the shared logic to `formatPlaceName()` in assets/js/explorer-utils.js (matching this file's existing extracted- pure-helper convention, unit-tested under Node) and replaced all 4 call sites. The new unit test reproduces the bug directly with a fake non-Array iterable shaped like an Arrow Vector, asserting `Array.isArray(vector) === false` before asserting `formatPlaceName` still handles it correctly — so a regression back to a bare `Array.isArray` check would be caught even without live parquet data. Verified end-to-end against the real corrected R2 file (fresh browser origin, to rule out ES-module cache artifacts from repeated same-session testing): table Place column shows real values ("Axial Seamount summit caldera" for IGSN:321000001), Date column shows real values ("2013-12-20"), CSV export matches. 0 console errors. 48/48 JS unit tests and 40/40 pipeline tests pass.
…nearby list Codex review of the previous commit: the samples-table and CSV render paths for place_name already went through escapeHtml()/CSV-quoting, but two other call sites — the in-map sample detail card (updateSampleCard) and the cluster-click "Nearby Samples" list — interpolate placeStr/desc directly into innerHTML with no escaping. Not a new defect (this code predates today), but my isamplesorg#311 fix is what makes it newly exploitable: place_name (and result_time, same reasoning) went from 100%-NULL in every prior deploy to carrying real, externally-sourced text (SESAR/OpenContext/ GEOME/Smithsonian aren't sanitized inputs) for the first time today. Wrapped placeStr/desc/sample.result_time in escapeHtml() at both sites. Left sample.label/pid unescaped in these same functions — that's a separate, pre-existing gap (labels have carried real data in every past deploy, not newly activated by this change) and out of scope for this commit; worth a follow-up issue. Verified: clicked a sample with real place_name data (IGSN:321000001, "Axial Seamount summit caldera") — card and nearby-list both render correctly, 0 console errors. 48/48 unit tests still pass.
rdhyee
added a commit
that referenced
this pull request
Jul 10, 2026
Root cause of #326 (search can't find "Axial seamount summit caldera" despite it showing in the table's Place column): the deployed isamples_202608_sample_facets_v3.parquet predates the #311 builder fix (merged in #318), so its place_name column is NULL for all 6,026,242 rows — while the table displays Place from samples_map_lite_v3, which DID get the fix in #319. The search scoring weights place_name (w=2), but every value it scanned was NULL: any sample findable only by place could never match. Fix mirrors #319 exactly: rebuilt sample_facets from the live prod 202608 wide (downloaded fresh; samp/samp_geo counts match the builder manifest) with the current build_frontend_derived.py, and uploaded as isamples_202608_sample_facets_v4.parquet — a NEW filename, never an overwrite (immutable-cache rule). Validation: - 6,026,242 rows (unchanged universe) - place_name coverage 2,263,648 (37.6% — identical to #319's lite numbers) - ZERO rows differ from v3 on any non-place column (pure column-content fix) - "axial seamount summit caldera" → 284 matches, exactly consistent with lite_v3's exact-phrase count (my earlier 805 was a broader substring) - spot: IGSN:321000001 → "[Axial Seamount summit caldera]" The v3 references remaining in comments describe the flat facets file generically and are left untouched to keep this diff one line. Fixes #326 (part 1: missing results). Part 2 (search hang) is being characterized separately — likely the #190 full-read fallback pulling the (now 69.4 MB) facets file cold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AjnkWb4HpuLeDbYfzmY3X9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Deploys the #311 pipeline fix (merged in #318 —
build_frontend_derived.py'ssampCTE now resolvesplace_name/result_timevia theSamplingEvent/SamplingSitegraph traversal instead of a dead read offMaterialSampleRecord) to actual production data.Data change
Rebuilt
samples_map_litefrom the samewide.parquetcurrently live in production — verified byte-identical (exact row counts + per-source min/max pid match againsthttps://data.isamples.org/isamples_202608_wide.parquet) before rebuilding, so this is a pure column-content fix, not a data-vintage change. Uploaded to R2 asisamples_202608_samples_map_lite_v3.parquet— a new filename, not an overwrite: the live_v2file is servedCache-Control: immutable, max-age=31536000, so overwriting it would leave stale/broken copies at CDN edges and in visitors' browsers for up to a year (same reasoning_v2itself was introduced under, over the unsuffixed name).Rebuild results:
place_namenow resolves for 2,263,648/6,026,242 (37.6%) located samples,result_timefor 5,937,692/6,026,242 (98.5%) — both were 0% before.explorer.qmd'slite_urlnow points at_v3.A second, latent bug this surfaced
Testing the corrected data against the real page (not just the raw parquet) found a genuine frontend bug invisible until today: 4 places in
explorer.qmdrenderedplace_nameviaArray.isArray(placeParts) && .... Observable'sDuckDBClientreturns ArrowLISTcolumns as an ArrowVector— iterable, has.length, but not a plain JSArray, soArray.isArray()isfalseon it. Every non-nullplace_namewas silently rendering blank. Masked in every prior deploy becauseplace_namewas 100%NULLbefore today's rebuild.Extracted to
formatPlaceName()inassets/js/explorer-utils.js(matching the file's existing pure-helper-extraction convention, unit-tested under Node — new test reproduces the bug with a fake non-Array iterable), replaced all 4 call sites.Codex review
Two rounds:
lite_url/formatPlaceNamechange): confirmedArray.from()is correct for the DuckDB-WASM value shapes involved, confirmed the immutable-cache/new-filename reasoning is sound, but flagged that 2 of the 4formatPlaceNamecall sites (the in-map sample detail card, the cluster-click "Nearby Samples" list) interpolate the result directly intoinnerHTMLwith no escaping — not a new defect, but newly exploitable now thatplace_name/result_timecarry real, externally-sourced text (SESAR/OpenContext/GEOME/Smithsonian aren't sanitized inputs) for the first time.escapeHtml()helper.label/pidare still unescaped in these same renderers — pre-existing, not newly activated by this change, flagged as separate follow-up cleanup.)Verification
node --test tests/unit/), 40/40 pipeline tests pass.assets/js/*.jsis served with a 10-minuteCache-Control: max-age=600, which repeatedly bit same-session re-testing during this work; a fresh context sidesteps that entirely and is what a real first-time visitor sees) againsthttps://rdhyee.github.io/isamplesorg.github.io/explorer.html: 0 console errors, samples table shows real Place ("Axial Seamount summit caldera") and Date ("2013-12-20") forIGSN:321000001, CSV export matches, and clicking that same sample renders correctly (escaped) in the detail card.🤖 Generated with Claude Code